Repository navigation
Conversation
Take `OffsetSizeBytes` by value in `unpack_u32_at_offset`, remove `unpack_u32` in favor of it, and extract `str_from_bytes` from `string_from_slice`. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Add `VariantView`, `VariantMetadataView`, `VariantObjectView` and `VariantListView`. Building a view reads one header. Each accessor reads and bounds-checks only the bytes it needs, and compares field names as bytes. `VariantMetadata`, `VariantObject` and `VariantList` are now built by promoting a view, and their accessors delegate to it. `ObjectBuilder::finish` sorts fields with the new `MetadataBuilder::cmp_field_names`. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`variant_get` reads each row of an unshredded column through `VariantView::get_path`. A row reads only the headers and field names along the path, and decodes only the value that it finds. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`shred_variant` and `unshred_variant` copy each leftover object field by its field id and encoded bytes, through the new `ObjectBuilder::insert_encoded` and `VariantView::bytes`. A leftover field is no longer decoded into a `Variant` and encoded again. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
This PR is stacked on #12308, #12309 and #12310. Its own change is the fourth commit, "perf(variant): copy leftover object fields by field id when shredding". It is a draft until they merge.
Rationale for this change
When
shred_variantandunshred_variantcopy an object field that is not shredded, they decode it into aVariant, look its name up in the metadata dictionary to get the field id back, and encode the value again. The output uses the same metadata dictionary as the input, so the field id and the encoded bytes are already correct.What changes are included in this PR?
ObjectBuilder::insert_encodedadds a field by its field id and its encoded bytes.VariantView::bytesreturns the exact encoding of a value.shred_variantandunshred_variantcopy leftover object fields through them. They match field names as bytes against a sorted list of the shredded field names.Are these changes tested?
The existing shredding tests cover the changed code.
VariantView::byteshas tests for every primitive type, and for the fields of an object and the elements of a list, whose encodings must tile the value region of their parent.Benchmarks (local run, this PR against #12310, x86_64 Linux):
shred_variant_partial_object_8k_rowsshred_variant_unmatched_object_8k_rowsunshred_variant_listunshred_variant_list_of_objectsunshred_variant_nested_fallbackunshred_variant_objectunshred_variant_partial_objectunshred_variant_partial_object_unsorted_metadataunshred_variant_primitivevariant_get_unshredded_object_path_262k_rowsvariant_get_list_index_unshreddedvariant_get_utf8_from_unshredded_stringvariant_get_binary_from_stringSlower inputs:
unshred_variant_objectis 4% slower andunshred_variant_listis 3% slower.variant_get_utf8_from_unshredded_stringandvariant_get_binary_from_stringare 9% slower in this run, although this PR does not changevariant_get; these two benchmarks have moved by 10–15% with unrelated changes on this machine.Are there any user-facing changes?
New public API in
parquet-variant:ObjectBuilder::insert_encodedandVariantView::bytes.On malformed variant bytes,
shred_variantand the leftover copy inunshred_variantreturn an error. Before, these paths could panic.🤖 Generated with Claude Code